Skip to content

Do not rerun a failed activity redispatched before its failure is recorded (#1241) - #1245

Merged
rmcdaniel merged 4 commits into
mainfrom
fix/flaky-exception-logging-replay-1241
Nov 19, 2026
Merged

rmcdaniel merged 4 commits into
mainfrom
fix/flaky-exception-logging-replay-1241

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Fixes #1241.

Cause: an engine defect, not a test race

ExceptionLoggingReplayTest::testSignalRetryLogsEachSequentialException recorded three exceptions because the second activity attempt really executed twice. The laravel-log-postgresql-12 artifact of the failing run (44554854139) shows first failure, second failure, second failure from that test. The test's expectation of two exceptions is correct.

A v1 activity's failure is recorded by a separately queued Workflow\Exception job. Laravel releases the activity's ShouldBeUnique lock when the job fails (CallQueuedHandler::failed releases it before calling Activity::failed(), which queues the Exception job). Until that Exception job writes the log at the activity's index, ActivityStub::make() sees no log and the unique lock is free. Any replay in that gap (a signal, timer or sibling result) dispatches the activity again. Activity::handle() only skipped when a log already existed, so the copy ran the activity again. That added an exception row and repeated the activity's side effects. In the flaky test, the Exception job's wake and the signal's wake both replayed the workflow, and the second replay landed in that gap.

Successful results have no such gap: ActivityMiddleware::onUnlock writes the log before the unique lock is released.

Fix

  • In its catch block, Activity::handle() stores the failing queue job's UUID under laravel-workflow-activity-failure:{workflow}:{index}. This runs before the worker can fail the job and release the unique lock.
  • Another job for the same index that finds the mark completes without executing. ActivityMiddleware checks the mark before ActivityStarted, so a skipped copy fires no activity events and records no result. Retries of the same job (same UUID) still execute.
  • The mark expires after workflows.activity_failure_mark_seconds (DW_V1_ACTIVITY_FAILURE_MARK_SECONDS, default 3600). If the Exception job is lost (a flushed queue during a deploy, an APP_KEY rotation that makes its payload undecryptable, failed() throwing before it dispatches, a manual delete), the mark expires and the next wake runs the activity again, as before this PR, instead of the workflow waiting forever. One hour is far above normal Exception-job latency: that is seconds, made of the 1 s lock and probe retries plus a wait behind earlier pending work in the same workflow. The setting is a top-level key so applications with an older published workflows.php, whose v1 array would shadow a nested key, still get the default.
  • Workflow\Exception removes the mark once it has recorded the failure or let it go: persisted, already logged, probe skip, or a workflow that is no longer running. While it is waiting (released), the mark stays.
  • When the logs were loaded with the job's payload, the activity rechecks them after reading the mark. This covers an Exception job that recorded the failure and removed the mark after that snapshot.
  • The skip decision is read once per handling, via Activity::awaitsRecordedFailure(), into @internal typed properties that are never serialized.
  • Known limit: a job that times out, or exhausts its attempts after a worker crash (Laravel Queue/Worker.php timeout and max-attempts paths), fails without running the activity's catch block. It writes no mark, so the race stays open on those paths, no worse than before. The CHANGELOG says so too.

Replay compatibility: recorded history, log/exception rows and job payloads are unchanged. The new flag is an uninitialized typed property, so it is never serialized. Jobs already in flight from older releases behave as before.

Evidence

  • New deterministic regression, FailedActivityRedispatchTest, uses --once workers on a private queue. It fails the activity's job, replays the workflow before the failure is recorded (the replay does dispatch the activity again), holds the Exception job, then runs the copy. Before the fix it failed 20/20 at "The failed activity ran again" (2 executions). After the fix it passed 20/20.
  • The original test, isolated, on PostgreSQL 17 + Redis 7 + PHP 8.3 in capped containers: before the fix, 0/60 quiet failures and 0/150 under in-container CPU stress. After the fix, 0/150 under stress. The CI flake depends on scheduling across two workers and did not reproduce locally, so the deterministic test above is the evidence for the cause.
  • testLostExceptionJobLeavesTheActivityRunnableAfterTheMarkExpires deletes the queued Exception job, runs a wake's copy while the mark lasts (it does not run), waits for a 2 s configured mark to expire, then wakes again. The activity runs a second time and the workflow completes. This is the reviewer's scenario (it stayed in WaitingStatus forever on the previous head), with expiry added.
  • New unit tests cover marking, a copy skipping without events (ActivityMiddlewareTest), the mark TTL setting, same-job retries clearing the mark, a released Exception job keeping it, and a recorded failure removing it. ExceptionLoggingReplayTest, ActivityCompletionWakeTest, ActivityDispatchBoundaryTest, ExceptionWorkflowTest and FailingWorkflowTest pass locally on PostgreSQL. ecs and phpstan pass on the changed files.
  • .github/feature-test-timings.json has weights for the new file. verify-feature-shards.php passes.

🤖 Generated with Claude Code

…orded

A v1 activity's failure is recorded by a separately queued
Workflow\Exception job, after Laravel has released the activity's unique
lock. A workflow replay in that gap dispatched the activity again and the
copy executed it, adding an exception row and repeating its side effects
(the #1241 flake: the second retry attempt ran twice).

The failing job now marks the activity in the cache before its lock is
released. Another job for the same index that finds the mark completes
without executing or recording a result; the Exception job removes the
mark once it has recorded the failure or let it go. History, payloads
and replay are unchanged.

Fixes #1241.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@rmcdaniel rmcdaniel added the agent:general Work owned by General label Nov 19, 2026
rmcdaniel and others added 3 commits November 19, 2026 16:58
…n unit tests

Track the Exception job's own release instead of asking the queue job,
expose the copy's skip as an unserialized public property, and cover
marking, skipping, same-job retries and removal in unit tests.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Review of #1245: a mark kept forever stranded a workflow whose
Workflow\Exception job was lost (flushed queue, undecryptable payload),
because every later copy of the activity skipped. The mark now expires
after workflows.activity_failure_mark_seconds
(DW_V1_ACTIVITY_FAILURE_MARK_SECONDS, default 3600); after that a wake
runs the activity again, as before. The key is top-level so applications
with an older published workflows.php receive the default.

ActivityMiddleware now checks the mark before ActivityStarted, so a
skipped copy fires no activity events. Adds the lost-Exception-job
regression and unit coverage; documents that timeout and
max-attempts-after-crash failures write no mark.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel merged commit aff2d33 into main Nov 19, 2026
41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:general Work owned by General

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: ExceptionLoggingReplayTest signal retry records three exceptions instead of two on PostgreSQL

1 participant